Systems and methods for generating a financial market snapshot

ABSTRACT

Systems and methods for generating a financial market snapshot. A plurality of financial market data elements can be received from a plurality of sources. The respective workflows can be determined using a plurality of workflows in a memory. A plurality of dependencies linking the respective workflows with one another can be determined. A financial market snapshot can be generated as a result of the processing of the workflows.

CROSS REFERENCE PARAGRAPH

This application is a continuation application of U.S. application Ser.No. 16/035,253, which was filed on Jul. 13, 2018, which is incorporatedby reference in its entirety for all purposes.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates a method for generating a financial market datasnapshot, according to an embodiment of the invention.

FIG. 2 illustrates a system diagram, according to an embodiment of theinvention.

FIG. 3 illustrates an example of how data can be used in creating asnapshot, according to an embodiment of the invention.

FIG. 4 illustrates a financial market data snapshot, according to anembodiment of the invention.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

Methods and systems are provided for generating an end-of-day snapshotthat may provide consistent information for financial market data frommultiple data sources. An end-of-day snapshot may comprise a stockquote, which may combine data from several sources and/or may includederived values based on data from multiple sources.

Example types of data that may be used may comprise:

-   -   Ticker symbol (i.e., MSFT)—May comprise the identifier that an        exchange issues and uses for a traded security.    -   Other industry identifiers—May comprise identifiers that other        data vendors have introduced to standardize across exchanges.        For example, ISIN and CUSIP from Standard & Poor's, SEDOL from        the London Stock Exchange, Valoren from SIX Financial        Information.    -   Closing price—May comprise the official closing price that an        exchange publishes.    -   Sector and industry classification—May comprise multiple data        vendors, such as FactSet and GICS, which categorize companies to        describe the line of business.    -   Revenue—May comprise revenue metrics that governments (e.g., US)        companies report to the SEC with their quarterly and annual        financial statements.    -   Price-to-earnings—May comprise a derived value based on the        latest trade price and earnings from the most recent financial        statement.    -   Corporate actions—May comprise events that affect the valuation        of a security and typically require adjusting historical prices.        For example, stock splits, spin-offs, and dividends.

Generating a snapshot of end-of-day prices for some or all securitiestraded on an exchange may require ingesting data from multiple datasources as well as changes that should be made to the snapshot based onthe data received from the multiple data sources (referred to as“dependencies”). For example:

-   -   Corporate actions—Can result in changes to a security's ticker        symbol and/or other industry identifiers. Can generate an        adjustment factor for historical prices prior to this date.    -   Earnings announcement—Can update financial statement metrics        such as revenue and earnings. May be required for derived values        such as price-to-earnings.    -   Sector and industry classifications—May reflect the most recent        classification for each security.    -   Closing price—May be the last trade executed that day, but can        be revised by an exchange after the trading session ended. May        be required for derived values such as price-to-earnings.

In some embodiments, the system can automate the generation of financialmarket data snapshots by efficiently processing market data frommultiple sources using a rules engine that represents theinterdependencies between different data sets. The system may use theconcept of a workflow (e.g., a multi-step workflow in some embodiments)to capture dependencies and automatically trigger a process when thesedependencies are fulfilled. Consistent snapshots in time for financialmarket data may be generated using one or more of the followingconcepts:

-   -   Rules-driven workflow configuration—A central repository can be        maintained for declaring the conditions that must be met for        each work item. Each workflow can comprise one or more work        items.    -   Multiple dependency constraints—One or more conditions that must        be satisfied can be defined for each work item. These conditions        may comprise: the completion of another work item, the existence        of a file, or a combination of completed work items and files.    -   Time-based conditions—Time-based conditions can be defined for a        work item. A time-based condition may be specific for a time        zone and it can be specified whether or not to adjust for        daylight savings time. Time-based conditions can specify a time        after which a work item can begin, as well as a time after which        a work item cannot begin. A work item may have time-based        conditions, dependency constraints, or both.    -   Automated process invocation—The processing of work items may be        automatically invoked when all dependency constraints and        time-based conditions are met.    -   Distributed, redundant work scheduler—A distributed architecture        for redundancy and scalability purposes may be used, allowing        multiple instances to evaluate dependency constraints and        time-based conditions. The distributed architecture may enable        scaling out to support higher processing loads, as well as        redundancy to avoid disruptions in service in the event that an        instance fails unexpectedly.    -   Automatic load balancing—A shared queue, just-in-time processing        model may be used. When an instance has available capacity, it        may check the shared queue for work items ready for processing.    -   Auto-recovery—Automatically retrying to execute a work item may        be done if it does not complete successfully.

FIG. 1 illustrates a method for generating a financial market datasnapshot, according to an embodiment of the invention. In 105, financialmarket data elements are received from a plurality of sources. In 110,workflows are determined, where each workflow processes a separatefinancial market data element. In 115, Dependencies linking workflowswith each other are determined. In 120, When one or more dependencyconditions are met, the linked workflows are processed. In 125, thefinancial market data snapshot is generated using processed workflows.

For example, in 105, generating snapshots of end-of-day prices on USexchanges may require incorporating data from the following datasources:

-   -   4 pm Eastern snapshot of last trade prices from NYSE and NASDAQ.    -   Revised closing prices within 2 hours 4 pm Eastern market close.    -   Daily security master files from Vendor 2    -   Daily files from Vendor 1 containing sector and industry        classifications.    -   Daily corporate action files from Vendor 3    -   SEC financial statement filings

In 110, the workflows can be determined. Each workflow can process aseparate financial market data element. In 115, dependencies linkingworkflows with each other are determined. In 120, when dependencies aremet, linked workflows can be processed. In 125, the financial marketdata snapshot, such as the one illustrated in FIG. 4 can be generated.

FIG. 2 illustrates an administrator 205 who can use the snapshotapplication to define and maintain workflows, which can be stored in thedefinitions database 210. Each workflow can identify at least one of thefollowing: (a) Conditions necessary to invoke a workflow (e.g., at leastone of: the existence of a file, a scheduled time, and/or a trigger);(b) Processes that the workflow invokes (e.g., at least one of: anexecutable, a registered subroutine, and/or a workflow); and (c)Auto-recovery options (e.g., at least one of: after a maximum number ofretries, upon failure, after a maximum allowable elapsed time, and/orafter not receiving a heartbeat for a maximum allowable elapsed time).

In some embodiments, the system itself can be defined as a workflow. Theadministrator 205 can configure the system by revising this workflow.For example, the workflow can comprise at least one of: a triggerprocess and/or a queue workers process. For example, the administrator205 can define one or more triggers (e.g., conditions being met thatindicate that all relevant data to generate the snapshot has beenreceived). The User Interface (UI) 207 can thus send a message to atrigger workers module 210 (e.g., in Data Center 2 (e.g., in Oregon))when necessary conditions are met to invoke one or more workflows. Forexample, if all the conditions have been met that indicate that all dataneeded to generate the snapshot has been received, the UI 207 can send amessage to the trigger workers module 210 indicating that the workerscan begin processing the workflows, Note that in some embodiments,different information needed to generate a particular workflow orseveral but not all workflows can be separately triggered withoutwaiting for all necessary information for the snapshot, such that fewerworkflows remain to be processed when the final information comes inthat is needed to provide a complete snapshot.

Once the trigger workers module 210 receives information indicating thatone or more workflows can be invoked (e.g., because the necessaryconditions have been met), the trigger workers module 210 can sendinformation to queue workers modules 215, which may be located inseveral data centers (e.g., in Data Centers 1 (e.g., in Virginia), 3(e.g., in Singapore), and 4 (e.g., in Ireland)), indicating that theworker modules can reach out to the queue database 220 to ask to beassigned workflows to process. In some embodiments, an auto-recoveryprocess can be invoked after a certain number of retries (e.g., three)to invoke the queue workers process.

The worker modules can access the definition database 225 as needed tocomplete their assigned workflows. Because the workflows are done inmultiple data centers, redundancy and distribution are built into themethod, so that if a data center is not functional, the snapshot canstill be triggered. In addition, because multiple data centers are usedto generate the snapshot, in some embodiments, no data centers willlikely be overwhelmed by the data processing required to generate thesnapshot. Although the trigger workers module 210 is shown in FIG. 2 asresiding at a different data center than the queue workers modules 215,in some embodiments, the trigger workers module(s) 210 and the queueworkers module(s) 215 can reside at the same data center

In some embodiments, the system can populate the queue database 220 withworkflows scheduled to run at specific times. The queue workers module210 can poll the queue database 220 for workflows that are ready forprocessing (e.g., based on a scheduled time that has passed). In someembodiments, only one queue worker can execute an available workflow,based on a first-come, first-serve model. A queue worker can do any orall of the following: poll the queue database 220 for an availableworkflow; validate the conditions necessary to invoke that workflow;lock the workflow in the queue database 220 for processing; invoke theprocess associated with that workflow; and/or update the queue database220 when the process has completed.

FIG. 3 illustrates an example of how data can be used in creating asnapshot, according to an embodiment. In 305, company, reference data(e.g., such as company, sector and/or industry files) can be used in anend-of-day reference data snapshot process to create an end of dayreference data snapshot 310. Company files can include files fromcompanies such as Microsoft, Google, Bank of America, etc. Sector filescan include files from a technology services sector, a finance servicessector, etc. Industry files can include files from a packaged softwareindustry, an Internet software or services industry, regional banks,etc.

In 315, US real-time data can be used in an end-of-day price snapshotprocess to generate an end-of-day price snapshot 320. For example,identifier information (e.g., MSFT, GOOG, BAC), close price information(e.g., 36.35, 1135.10, 23.03) and/or volume information (e.g., 35369170,4158000, 116139909) can be used for each of the companies Microsoft®,Google®, and Bank of America®.

In 340, sector master file information can be used in an end-of-daymaster snapshot process to create an end-of-day master snapshot 345. Forexample, identifier information (e.g., MSFT, GOOG, GOOGL, BAC), companyname (e.g., Microsoft®, Google®, Google®, Bank of America®), and/orissue information (e.g., class A, class C, class A, class A) can beused.

In 360, corporate action files can be used in an end-of-day corporateaction snapshot process to create an end-of-day corporate actionsnapshot 355. For example, identifier information (e.g., GOOG, GOOG,GOOGCB, BAC), action information (e.g., spin-off, rename, rename,dividend), value (e.g., 1, 1, 1, 0.01), and/or new identifierinformation (e.g, GOOGV, GOOGL, GOOOG) can be used.

In 370, SEC filing information can be used in an end-of-day SEC snapshotprocess 370 to create an end-of-day SEC snapshot 365. For example,company name (e.g., Microsoft®, Google®, Bank of America®), revenueinformation (77654, 59730, 97773), earnings information (e.g., 21486,12056, 953280) and/or outstanding share information (e.g., 8328, 671,10592) can be utilized.

FIG. 4 illustrates an excerpt of an end-of-day snapshot file for NYSEsecurities:

-   -   Date—From the End of day price snapshot.    -   Symbol—Actual symbols from the US real-time data feed are        translated to the Symbols in this file from the End of day        master snapshot.    -   Exchange—From a source not shown in the workflow diagram—An        industry-standard exchange code from any source from a lookup        table for exchanges.    -   Currency/DividendCurrency—From a source not shown in the        workflow—we assign an industry-standard currency code from a        lookup table for exchanges.    -   Open/High/Low/Last/Volume—From the Adjusted price snapshot.    -   SplitRatio/CashDividend—From the End of day corporate action        snapshot.

The end of day reference data snapshot 310 can be used in an adjustedquote workflow 390. The end-of-day price snapshot 320 can be used in anadjusted price workflow 325. The end-of-day master snapshot 345 and theend-of-day corporate action snapshot 355 can be used in a securitylinking workflow 350. The end-of-day SEC snapshot can be used in anadjusted derived data snapshot 375.

Example Use Case

Three components may be included in embodiments of the invention:Database, MessageBus and FlowWorkers. Database can containWorkFlowDefinitions and individual WorkFlowItems. MessageBus cancomprise a generic message bus for FlowWorkers to send/receive WorkFlow.FlowWorkers can comprise a property partition which can define whichpartition of MessageBus it can pick up WorkFlows from. A FlowWorker canpick up a WorkFlow of the same Partition from MessageBus, make itinvisible to other FlowWorkers and work on it.

MessageBus. MessageBus can be any generic message bus that supportsmultiple publishers/subscribers and message partitions.

Database. Database can be any database. A table can be created forWorkFlowDefinitions and/or for WorkFlowItems. For example, the table forWorkFlowDefinitions can be:

Columns:

ID (Can be a unique ID of the WorkFlowDefinition) Name Friendly name ofthe WorkFlowDefinition IsEnabled Is enabled DateScheduled Start dateTimeZone Time zone TimeDelay Start time Frequency Recurring WorkFlowfrequency in seconds. 0 means one time WorkFlow. Partition MessageBuspartition TimeSpanWindow The time range to schedule WorkFlows. Forexample, from 3 days ago to now. MaxAttempts Maximum number of attemptsbefore giving up. Timeout Timeout value of each WorkFlow execution.RetryWaitPeriod Wait time before next retry. WorkflowType Scheduler,EndOfDay Parameters Parameters of the Workflow Dependencies Theexistence of a file or the completion of another WorkFlow. VersionVersion of the WorkFlowDefinitionThe table for WorkFlowItems can be:

Columns:

ID (Unique ID of the Workstem) WorkDefinitionId Id of WorkFlowDefinitionTimestampScheduled The scheduled time for the WorkItemTimestampSubmitted The time the WorkItem is created TimestampStart Thetime when a FlowWorker starts working on the WorkItem TimestampEnd Thetime when a FlowWorker finishes working on the WorkItem State Scheduled,Started or Completed Result None, Success, Failure or Aborted AttemptThe number of attempts Message Any output message from FlowWorkerEach FlowWorker can do the following:

-   -   1. Pick up a WorkFlow of the same Partition from MessageBus.    -   2. Use the WorkFlow ID to get corresponding WorkFlowItem from        Database.    -   3. Use the WorkFlowItem.WorkDefinitionId to get corresponding        WorkFlowDefinition from Database.    -   4. Perform actual work based on WorkFlowDefinition.    -   5. Update WorkFlowItem (State, Result, etc) in Database.        The actual work (#4) can vary for different types of WorkFlows.        The following examples of FlowWorkers demonstrates how        XigniteFlow can be used in some embodiments.

Scheduler FlowWorker. Scheduler FlowWorker can be a separate tool thatcan add a new Scheduler WorkFlow to MessageBus periodically. In someembodiments, it can perform some or all of the following procedures:

-   -   1. Pick up a Scheduler WorkFlow from MessageBus.    -   2. Use the WorkFlow ID to get corresponding WorkFlowItem from        Database.    -   3. Use the WorkFlowItem.WorkDefinitionId to get corresponding        WorkFlowDefinition from Database.    -   4. Perform actual work which is to manage WorkFlowItems for all        other WorkFlows.        -   a. For each WorkFlowDefinition, create WorkFlowItem if all            the following conditions are met.            -   i. The TimeSpanWindow contains scheduled time to run                (determined by WorkFlowDefinition DateScheduled,                TimeDelay, TimeZone and Frequency) but WorkFlowItem for                that scheduled time has not been created yet.            -   ii. For each defined dependency on a file, the file                exists.            -   iii. For each defined dependency on another WorkFlow                schedule to run at certain time, the WorkFlow has been                completed successfully.        -   b. For each WorkFlowItem with TimestampScheduled in a            configured range, State Started and the elapsed time between            WorkFlowItem.TimeStampStart and now is more than            WorkFlowDefinition.Timeout, set the WorkFlowItem State to            Completed and Result to Aborted in Database. Create a new            WorkFlowItem with same configuration and Attempt incremented            by 1.        -   c. For each WorkFlowItem with TimestampScheduled in a            configured range, State Completed and Result Failure, create            a new WorkFlowItem with same configuration and Attempt            incremented by 1.        -   d. For each created WorkFlowItem, add a WorkFlow message            with the WorkFlowItem Id to corresponding MessageBus            partition.    -   5. Update the current Scheduler WorkFlowItem as Completed with        Success.

Single-source Snapshot FlowWorker. Single-source Snapshot FlowWorker cangenerate single-source snapshot files such as an end-of-day referencedata snapshot, an end-of-day price snapshot, an end-of-day mastersnapshot, an end-of-day corporate action snapshot, an end-of-day SECsnapshot, etc. In some embodiments, it can perform some or all of thefollowing procedures:

-   -   1. Pick up a Single-source Snapshot WorkFlow from MessageBus.    -   2. Use the WorkFlow ID to get corresponding WorkFlowItem from        Database.    -   3. Use the WorkFlowItem.WorkDefinitionId to get corresponding        WorkFlowDefinition from Database.    -   4. Perform actual work which is to generate single-source        snapshot file based on WorkFlowDefinition and Parameters,        including        -   a. Source of data (realtime price feed, security master            database, corporate action database, etc)        -   b. How to get the data (stored procedure or service call)        -   c. Which exchange        -   d. Format of output snapshot file        -   e. Location of output snapshot file    -   5. Update the current WorkFlowItem (State, Result, Message, etc)        in Database.

Multi-source Snapshot FlowWorker. Multi-source Snapshot FlowWorker cangenerate multi-source snapshot files using a combination ofsingle-source snapshots. In some embodiments, it can perform some or allof the following procedures:

-   -   1. Pick up a multi-source Snapshot WorkFlow from MessageBus.    -   2. Use the WorkFlow ID to get corresponding WorkFlowItem from        Database.    -   3. Use the WorkFlowItem.WorkDefinitionId to get corresponding        WorkFlowDefinition from Database.    -   4. Perform actual work which is to generate Multi-source        Snapshot file based on WorkFlowDefinition and Parameters,        including        -   a. The Single-source snapshot files to use        -   b. The logic to combine multiple single-source snapshot            files        -   c. Which exchange        -   d. Format of output snapshot file        -   e. Location of output snapshot file    -   5. Update the current WorkFlowItem (State, Result, Message, etc)        in Database.

CONCLUSION

While the disclosure has been described with reference to numerousspecific details, one of ordinary skill in the art will recognize thatthe disclosure can be embodied in other specific forms without departingfrom the spirit of the disclosure. In addition, a number of the figuresillustrate processes. The specific operations of these processes may notbe performed in the exact order shown and described. The specificoperations may not be performed in one continuous series of operations,and different specific operations may be performed in differentembodiments. Thus, one of ordinary skill in the art would understandthat the invention is not to be limited by the illustrative details, butrather is to be defined by the appended claims.

While various embodiments have been described above, it should beunderstood that they have been presented by way of example and notlimitation. It will be apparent to persons skilled in the relevantart(s) that various changes in form and detail may be made thereinwithout departing from the spirit and scope. In fact, after reading theabove description, it will be apparent to one skilled in the relevantart(s) how to implement alternative embodiments. Thus, the presentembodiments should not be limited by any of the above-describedembodiments.

In addition, it should be understood that any figures which highlightthe functionality and advantages are presented for example purposesonly. The disclosed methodology and system are each sufficientlyflexible and configurable such that they may be utilized in ways otherthan that shown.

Further, the purpose of any Abstract of the Disclosure is to enable theU.S. Patent and Trademark Office and the public generally, andespecially the scientists, engineers and practitioners in the art whoare not familiar with patent or legal terms or phraseology, to determinequickly from a cursory inspection the nature and essence of thetechnical disclosure of the application. An Abstract of the Disclosureis not intended to be limiting as to the scope of the present inventionin any way.

Although the term “at least one” may often be used in the specification,claims and drawings, the terms “a”, “an”, “the”, “said”, etc. alsosignify “at least one” or “the at least one” in the specification,claims and drawings.

Additionally, the terms “including”, “comprising” or similar terms inthe specification, claims and drawings should be interpreted as meaning“including, but not limited to.”

Finally, it is the applicant's intent that only claims that include theexpress language “means for” or “step for” be interpreted under 35U.S.C. 212, paragraph 6. Claims that do not expressly include the phrase“means for” or “step for” are not to be interpreted under 35 U.S.C. 212,paragraph 6.

What is claimed is:
 1. A method, comprising: receiving, on at least oneprocessor, a plurality of financial market data elements from aplurality of sources; determining, with the at least one processor,respective workflows using a plurality of workflows in a memory;determining, with the at least one processor, a plurality ofdependencies linking the respective workflows with one another; andgenerating, with the at least one processor, a financial market snapshotas a result of the processing of the workflows.